The Error-reject Tradeoff

We investigate the error versus reject tradeo for classi ers. Our analysis is motivated by the remarkable similarity in error-reject tradeo curves for widely di ering algorithms classifying handwritten characters. We present the data in a new scaled version that makes this universal character particularly evident. Based on Chow's theory of the error-reject tradeo and its underlying Bayesian analysis we argue that such universality is in fact to be expected for general classi cation problems. Furthermore, we extend Chow's theory to classi ers working from nite samples on a broad, albeit limited, class of problems. The problems we consider are e ectively binary, i.e., classi cation problems for which almost all inputs involve a choice between the right classi cation and at most one predominant alternative. We show that for such problems at most half of the initially rejected inputs would have been erroneously classi ed. We show further that such problems arise naturally as small perturbations of the PAC model for large training sets. The perturbed model leads us to conclude that the dominant source of error comes from pairwise overlapping categories. For in nite training sets, the overlap is due to noise and/or poor preprocessing. For nite training sets there is an additional contribution from the inevitable displacement of the decision boundaries due to niteness of the sample. In either case, a rejection mechanism which rejects inputs in a shell surrounding the decision boundaries leads to a universal form for the error-reject tradeo . Finally we analyze a speci c reject mechanism based on the extent of consensus among an ensemble of classi ers. For the ensemble reject mechanism we nd an analytic expression for the errorreject tradeo based on a maximum entropy estimate of the problem di culty distribution.